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FOREWORD 


This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  reading  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  in  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of. 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 
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T&E  GUIDELINES  FOR  ASW  SYSTEMS 


The  checklist  items  presented  here  are  specifically  applicable  to  ASU 
testing  and  evaluation.  It  is  suggested  that  the  user  of  this  volume  also 
refer  to  the  Report  of  the  Defense  Science  Board  on  Test  and  Evaluation 
which  contains  general  checklist  items  also  applicable  to  this  system  T&E 
program. 

In  the  utilization  of  checklists  on  ASW  programs,  due  regard  must  be 
given  to  the  fact  that  ASW  encompasses  the  use  of  a  great  many  different 
kinds  of  systems  and  subsystems.  A  partial  list  follows: 

Sensors 

Acoustic,  both  active  and  passive 

Arrays 

Sonobuoys 

Sonars 

Electronic 

! 

Radar 
ESM 

Magnetic 

Platforms 

Surface  Ships 
Submarines 
Fixed  wing  aircraft 
Helicopters 

Command  and  Control  Systems 

Weapons 

Depth  Charges 
Rocket-thrown  torpedoes 
Air  launched  torpedoes 
Ship  and  submarine  launched  torpedoes. 
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Where  possible,  the  particular  class  of  system/ subsystem  to  which  the 
statement  or  item  is  applicable  has  been  specified.  However,  in  cases  where 
not  specified,  reader  interpretation  is  expected.  Also,  the  reader  should 
review  other  specific  weapon  checklists  as  appropriate  to  the  ASW  program 
under  consideration  (i.e.,  aircraft,  ships,  etc.). 

The  checklist  items  presented  here  are  organized  into  time  phases  of 
the  acquisition  process  oriented  to  the  DSARC  cycle.  The  checklists  cover 
various  aspects  of  the  major  activities  that  should  be  underway  during  a 
given  time  period.  Hence,  a  checklist  might  cover  the  (1)  evaluation  of 
work  that  occurred  in  the  previous  phase,  (2)  conduct  of  testi'  planned  in 
in  the  previous  phase  and  executed  in  the  subject  phase,  and  (3)  plans  and 
other  preparatory  actions  for  test  schedules  to  be  conducted  in  a  subsequent 
phase.  For  reasons  such  as  this,  items  on  some  subjects,  such  as  develop¬ 
ment  test  plans,  may  appear  in  more  than  one  phase.  In  addition,  since  the 
Services  and  the  DSARC  have  flexibility  in  deciding  how  rapidly  to  progress 
in  the  validation  phase,  there  may  be  cases  where  the  Request  for  Proposals 
(RFPs) ,  proposal  evaluations,  source  selections,  or  contract  negotiations 
may  occur  after  the  DSARC  approves  full-scale  development  instead  of  before. 
For  this  reason,  it  is  recommended  that  previous  checklists  in  the  Validation 
Phase  be  reviewed  when  entering  the  Full-Scale  Engineering  Development  Phase. 
The  following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceptual  Phase  of  the  acquisition  of  the  system. 

This  phase  (of  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  development  of  a  weapons  system  con¬ 
cept  that  offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  out  in  DoD  Directive  5000.1  specifically,  the 
objectives  of  this  phase  should  be: 

1.  To  verify  that  there  is  a  military  need  for  the  proposed  system. 

2.  To  determine  whether  or  not  there  is  a  sound  physical  basis  for 
a  new  weapons  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical  phenomena, 
for  satisfying  the  military  need. 
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4.  To  determine  whether  or  not  the  proposed  solution  is  superior  to 
its  competitors  in  terms  of  potential  effectiveness,  probability 
of  success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  needs  to  deter¬ 
mine  whether  or  not  it  is  better  to  start  advanced  developments 
now  rather  than  to  wait  for  future  technological  improvements. 

6.  To  identify  the  key  risk  areas  and  critical  issues  that  need  to 
be  resolved  before  full-scale  development  is  initiated. 

The  most  important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  time  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSARC  recommends  full- 
scale  engineering  development  of  the  system) . 

While  these  objectives  are  not  spelled  out  in  the  DoD  Directive  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm: 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of  devel¬ 
opment  costs,  preliminary  estimates  of  life  cycle  costs  and  poten¬ 
tial  benefits  in  context  with  overall  DoD  strategy  and  fiscal 
guidance . 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  and  solutions  are  in 
hand. 

4.  Realism  of  the  plan  for  full-scale  development. 

In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&E  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the  T&E 
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interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must  include 
adequate  guidance  to  ensure  that  sufficient  resources  and  time  are  available 
so  that  engineering  effort  can  properly  support  the  initial  DT&E  with  hard¬ 
ware,  software,  technical  data,  and  training. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  4  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  incorporated  in  the  engineering  development  contract  as  well  as  the  DT&E 
associated  with  addressing  the  critical  issues  and  areas  of  major  risk 
identified  in  the  DCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  produc¬ 
tion  decision.  By  this  time,  the  system  is  well-defined  and  is  becoming  a 
unique  item  and,  hence,  sound  judgment  must  be  applied  in  using  these 
checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have  - 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be,  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development  process 
is  complete  and  that  the  design  risks  have  been  minimized  (the  sys¬ 
tem  is  ready  for  production) . 

2.  Demonstrate  that  the  system  will  meet  design  specifications. 

The  primary  objectives  of  the  IOT&E  should  be  to: 

3.  Assess  operational  suitability  and  effectiveness. 

4.  Validate  organizational  and  employment  concepts. 

5.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  production/development 
will  be  recommended  to  the  Secretary  of  Defense. 
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The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The  IOT&E 
plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed  to  evalu¬ 
ate  the  inputs  associated  with  the  primary  objectives.  These  tests  should 
not  be  conducted  until  the  primary  objectives  of  the  DT&E  have  been  met. 
Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E  major  milestone — 
the  system  is  ready  for  production — has  been  achieved.  Close  monitoring  of 
the  T&E  Service  activity  is  required  during  the  latter  stages  of  this  phase. 

SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the  early 
production  items. 

To  enter  the  Production/Deployment  Phase,  the  DSARC  will  have  reviewed 
the  program  to  confirm: 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of  pro¬ 
duction  and  logistic  problems  is  complete. 

•  The  realism  of  the  plan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 

2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the  previous 
phase;  however,  much  of  the  testing  is  verification  that  the  production  sys¬ 
tem  performance  is  as  expected.  Hence,  most  of  the  items  in  the  previous 
phase  are  appropriate  to  this  phase,  especially  those  related  to  OT&E. 
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I .  CONCEPTUAL  PHASE 

Activity  during  the  Conceptual  Phase  is  generally  concerned  with  the 
questions,  "How  does  this  system  differ  from  its  predecessor?"  "What  fea¬ 
tures  will  make  it  perform  better?"  Critical  issues  will  generally  relate 
to  these  new  features. 

Advances  in  ASW  acoustic  surveillance  are  being  accomplished  through 
the  use  of  lower  frequency  arrays  which  use  larger  and/or  an  increased  num¬ 
ber  of  hydrophones.  Important  T&E  issues  involve  the  physical  size  of  any 
new  array,  and  tests  should  demonstrate  its  ability  to  fit  selected  plat¬ 
forms,  both  new  and  older  ones  intended  for  backfit.  Demonstrations  should 
be  planned  to  test  their  ability  to  present  coherent  signals  and  resolve 
directional  ambiguities.  Effect  of  sea  pressure  on  new  components  should 
not  degrade  the  system.  New  features  should  not  lessen  reliability  and 
maintainability.  T&E  should  be  planned  to  show  adaptability  of  new  arrays 
to  a  spectrum  of  threat  noises,  including  the  future  and  potentially  quieter 
ones,  not  just  the  current  and  most  pressing  ones.  Plans  for  measuring  the 
important  acoustic  parameters  should  be  evaluated.  Even  at  this  early  state, 
performance  goals  should  be  specified  in  operational,  as  well  as  technical 
terms. 

ASW  effectiveness  is  dependent  upon  the  processing  of  data  at  a  cen¬ 
tral  control  point  where  the  data  from  many  sensors  are  correlated.  An 
examination  of  the  data  handling  capabilities  of  the  new  system  should  in¬ 
clude  security,  resistance  to  jamming,  and  degree  and  effectiveness  of 
automation.  IOT&E  should  be  planned  to  show  how  well  the  data  displays  are 
adapted  to  expected  operator  abilities  and  to  the  mission  requirements. 

Conceptual  Phase  Checklist  items  are  as  follows: 

1.  Use  of  Prototypes 

2.  Performance  Characteristics 

3.  Measurements  of  Mission  Effectiveness 

4.  ASW  Evaluation  Criteria 

5.  OT&E  Planning 
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6.  Facilities  and  Instrumentation  Requirements 

7.  Dependency  of  Production  Decision  on  Testing 

8.  Evaluation  of  Interfaces 

9.  Exploratory  Test  Results 

10.  Arrays 

11.  Effect  of  Changes  in  Threat  Noise 

12.  Operational  vs.  Technical  Performance 

13.  Reliability  and  Maintainability 

14.  Data  Handling 

15.  Automated  Detection  Schemes 
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USE  OF  PROTOTYPES 


If  the  acoustic  sensor  or  the  processor  subsystem  encompasses  new  con¬ 
cepts  or  technologies  for  detection,  localization  and  tracking  of  a 
target,  and  these  concepts/technologies  have  not  been  proven,  then  the 
advanced  development  testing  program  should  be  structured  around  the 
use  of  a  prototype  of  each  approach  designed  to  prove  the  system  con¬ 
cept  under  realistic  operational  conditions. 

At  least  one  such  approach  must  be  demonstrated  before  proceeding  to 
engineering  development.  When  an  implied  commitment  to  production  is  in¬ 
volved,  the  technology  should  be  operationally  proof  tested  prior  to  com¬ 
mencing  full-scale  development.  In  s  '.ase  where  this  was  not  followed,  a 
production  contract  was  negotiated  for  rardware  that  had  never  been  fully 
tested,  resulting  in  expensive  modifications  and  schedule  delays. 

2.  PERFORMANCE  CHARACTERISTICS 

Analytic  and  empirical  studies  should  be  conducted  prior  to  DSARC  I 
to  insure  that  the  range  of  critical  performance  characteristics  has 
been  specified. 

Each  performance  characteristic  so  specified  should  be  measurable 
through  bench  and  laboratory  testing.  The  test  design  and  the  number  of 
tests  should  be  adequate  to  provide  results  with  confidence  limits  compat¬ 
ible  with  the  statements  of  desired  characteristics.  For  example,  if  the 
measurement  of  the  minimum  discernible  signal  of  an  acoustic  sensor  system 
is  performed  with  a  device  whose  error  rate  is  large  (as  measured  by  stan¬ 
dard  deviations)  compared  to  the  expected  minimum  signal,  then  a  large 
number  of  tests  will  be  required  so  that  the  average  minimum  discernible 
signal  can  be  estimated  with  the  confidence  so  stated.  Testing  in  advanced 
development  should  be  planned  to  explore  the  performance  characteristics 
over  a  broad  range  of  system  inputs  so  as  to  provide  insight  .into  system 
performance  over  the  expected  range  of  values  and  not  just  at  a  single 
point.  For  example,  the  effects  of  sea  state,  time  of  year,  wind,  target 
type  and  line  spectrum  should  be  investigated. 
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MEASUREMENTS  OF  MISSION  EFFECTIVENESS 


During  the  conceptual  phase  of  the  acquisition  of  a  new  class  of  ASW 
ship,  a  study  effort  should  be  commenced  jointly  by  the  CNO  and 
COMOPTEVFOR  to  establish  mission-related  measures  of  effectiveness 
which  may  be  expressed  in  numerical  fashion  and  which  may  later  be 
made  the  subject  of  OT&E  to  determine  how  closely  the  new  ship  system 
meets  the  operational  need  for  which  it  was  conceived. 

As  an  example,  for  a  surface  ship  or  submarine  whose  primary  mission  is 
to  be  ASW,  threat-related  parameters  can  be  established  based  on  known 
adversary  capabilities;  own  ship  parameters  can  be  set  according  to  the 
capabilities  of  present  equipment  or  those  demanded  of  equipment  under 
development;  and  environmental  factors  can  be  entered  using  results  of  past 
and  present  investigations.  Based  on  such  study,  a  numerical  probability 
of  success  can  be  generated  in  each  of  the  mission  areas  conceived  for  the 
ship,  after  which  acceptable  minimum  standards  can  be  established  as  test 
criteria  for  operational  suitability.  Advantages  of  this  process  are  that 
it  will  serve  to  eliminate  bias  and  subjectivity  from  the  analysis  of  test 
results,  thereby  increasing  their  credibility,  and  secondly,  it  will  allow 
greater  use  of  computer  simulation  and  modeling  of  the  operational  scenario 
with  attendant  reduction  in  the  amount  of  live  testing. 

Analysis  and  tests  of  this  type  were  not  generally  conducted  in  past 
ASW  platform  developments,  resulting  in  the  deployment  of  systems  without 
hard  and  fast  mission  success  criteria  having  been  established.  Over  the 
years  expensive  backfits  and  modifications  were  necessary  to  bring  systems 
to  an  adequate  state  of  mission  effectiveness. 

4.  ASW  EVALUATION  CRITERIA 

The  test  and  evaluation  plan  should  include  the  evaluation  criteria 
to  be  used  for  the  selection  of  the  final  system  design. 

They  should  be  based  on  performance  factors  which  are  measurable 
through  testing  such  as  probability  of  detection  of  submarines  as  a  func¬ 
tion  of  range  and  attack  conditions  such  as  sea  state,  sound  velocity 
profile,  water  depth,  bottom  conditions  and  ocean  area;  effect  of  other 
shipping  and  noise  sources  in  area;  probability  of  false  alarms  under 
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various  scenarios  such  as  escort  mission,  ASW  barrier,  convoy  operation, 
or  operator  time  on  station;  probability  of  localization  given  a  detection, 
as  a  function  of  search  time  and  expended  resources  (sonobuoys,  MAD  runs), 
time  late  at  datum,  accuracy  of  datum,  or  convergence  zone  effect.  A  data 
collection  and  evaluation  plan  should  be  developed  which  describes  the 
range  of  acceptable  performance  for  each  factor. 

5.  OT&E  PLANNING 

Before  DSARC  I,  the  nature  of  the  schedule  for  the  IOT&E  plan  should 

be  addressed. 

The  IOT&E  plan  should  include  ASW  engagements  in  the  environments  in 
which  the  new  system  is  expected  to  operate.  ASW  testing  may  be  addressed 
in  several  phases,  such  as: 

(a)  One-on-one  testing  against  existing  U.S.  submarines  and  avail¬ 
able  simulators  of  the  assumed  threat. 

(b)  Multiple  ASW  vehicle  testing  in  a  multiple  ship  environment 
including  multiple  submarines  and  merchant  shipping. 

(c)  Comnarative  testing  (side-by-side  if  practicable)  of  the  ASW 
system  with  existing  systems  to  estimate  the  increased  capa¬ 
bility. 


6.  FACILITIES  AND  INSTRUMENTATION  REQUIREMENTS 

Before  DSARC  I  the  testing  facilities  and  instrumentation  requirements 
to  conduct  developmental  and  operational  tests  should  be  identified, 
along  with  a  tentative  schedule  of  test  activities. 

The  applicability  of  the  test  ranges  and  the  adequacy  of  the  facili¬ 
ties  and  instrumentation  should  be  verified  (for  example,  factors  such  as 
location  of  range  relative  to  submarine  operating  base,  location  relative 
to  weapon  support  facilities,  geographical  size  of  range  needed  for  scen¬ 
ario,  weather  expected  at  the  range  at  time  of  test).  Due  regard  should 
be  given  to  the  degree  that  the  range  compares  to  expected  operating  areas 
(e.g.,  AUTEC  does  not  adequately  simulate  the  Mediterranean).  Insofar  as 
possible  alternative  approaches  (different  ranges,  etc.)  and  instrumentation 
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improvements  needed  should  be  specified.  If  the  range  and  instrumentation 
factors  are  found  to  cast  significant  doubt  on  the  meaningfulness  of  the 
test  data  the  steps  necessary  to  assure  meaningful  data  should  be  identified 
and  planned. 

7.  DEPENDENCY  OF  PRODUCTION  DECISION  ON  TESTING 

Allow  for  sufficient  time  between  the  planned  end  of  demonstration 
testing  and  major  procurement  decisions  so  that  there  is  a  flexibility 
for  modification  of  plans  which  may  be  required  during  the  test  phases 
of  the  program. 

In  the  case  of  several  ASW  sensor  systems,  production  decisions  and 
delivery  schedules  were  dictated  by  previous  decisions  and  schedules 
regarding  platforms.  Thus,  for  these  systems,  production  decisions  were 
made  prior  to  completion  of  development.  In  addition,  this  constraint 
precluded  incorporation  of  modifications  resulting  from  testing. 

8.  EVALUATION  OF  INTERFACES 

Whenever  two  major  systems  are  to  be  connected,  the  interface  should 
be  carefully  evaluated  and  monitored  throughout  development. 

A  relatively  minor  change  in  one  of  the  systems  could  affect  the 
operation  of  the  other.  As  an  example,  minor  modification  to  an  ASW 
helicopter  has  rendered  it  physically  unsuitable,  because  of  increased 
size,  for  use  on  certain  ships. 

9.  EXPLORATORY  TEST  RESULTS 

The  use  of  brassboard  or  modified  existing  hardware  to  "prove"  that 
the  concept  will  work  should  be  seriously  scrutinized  to  ensure  that 
the  demonstration  and  tests  are  applicable. 

Results  of  tests  conducteu  during  exploratory  development,  which  are 
used  to  establish  the  feasibility  of  the  concept  or  technology,  and  which 
most  likely  have  been  conducted  on  brassboard,  breadboard,  or  modified 
existing  hardware  should  be  evaluated  with  special  attention  to  items 
such  as: 
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(a)  The  packaging  of  the  hardware  and  its  location  relative  to  other 
hardware  may  significantly  affect  the  performance  characteris¬ 
tics  so  that  the  suggested  proof  of  feasibility  is  inconclusive. 
This  is  especially  true  relative  to  the  electronic  package  on 
ASW  aircraft  or  remote  sensor  subsystems  such  as  sonobuoys.  As 
an  example,  the  physical  location  aboard  ships  of  particular 
subsystems  of  an  ASW  combat  system  and  the  manner  of  their  inter¬ 
connection,  is  proving  to  be  particularly  critical  to  operational 
effectiveness. 

(b)  The  environment  in  which  the  hardware  was  tested  may  preclude  the 
generation  of  data  needed  to  prove  that  the  approach  will  be 
applicable  to  an  operational  environment.  For  example,  the 
eifects  of  multi-layer,  convergence  zones  and  casual  friendly 
shipping  cannot  be  evaluated  unless  the  laboratory  is  situated 

in  such  an  ocean  environment. 

(c)  The  tests  must  include  noise  sources  representative  of  those 
that  might  be  expected  in  an  operational  environment.  If  the 
sensor  performance  is  sensitive  to  broad  band  noise  and  its 
internal  line  structure,  any  test  which  uses  artificial  noise 
sources  and  "clean"  target  signals  would  not  be  meaningful.  Tape 
recordings  obtained  under  operational  conditions  should  be  used, 
and  should  include  conditions  wherein  convergence  zone  and  bottom 
bounce  propagation  exist. 


10.  ARRAYS 

Since  there  is  a  trend  toward  lower  frequency  arrays  for  tactical  as 
well  as  surveillance  arrays,  the  size  and/or  type  of  hydrophone  used 
in  the  array  field  must  be  critically  analyzed. 

The  proposed  array  must  fit  in  the  intended  platform  without  introduc¬ 
ing  constraints.  It  should  be  capable  of  backfit  into  other  (older)  plat¬ 
forms.  If  new  mounting  or  towing  techniques  will  be  required,  they  should 
be  included  in  the  test  plan.  If  there  are  new  fabrication  techniques 
required,  a  demonstration  that  they  will  withstand  the  sea  pressure  at 
operating  depths  should  be  provided.  For  example,  in  the  case  of  towed 
arrays,  longer  apertures  are  being  developed,  and  test  thus  must  be  design¬ 
ed  to  evaluate  streaming  and  recovery  techniques  and  equipment,  strumming 
and  other  self-noise  effects,  depth  control,  and  maneuvering  constraints 
imposed  on  the  towing  vessel. 
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11.  EFFECT  OF  CHANGES  IN  THREAT  NOISE 


The  effectiveness  of  arrays  should  be  tested  using  various  types  of 
threat  noise  sources. 

An  array  is  generally  optimized  for  response  over  the  relatively 
narrow  frequency  range  which  contains  the  expected  threat  noise  source. 

It  is  important  to  assess  the  risk  if  the  threat  noise  source  is  replaced 
by  another  (or  if  the  characteristics  of  the  threat  noise  source  are 
changed).  As  an  example,  surveillance  arrays  in  broad  use  have  been 
optimized  for  particular  noise  sources.  Adequate  demonstrations  have  not 
always  been  conducted  of  their  effectiveness  under  conditions  involving 
changes  to  the  threat  noise  source. 

12.  OPERATIONAL  VS  TECHNICAL  PERFORMANCE 

Test  plans  should  be  formulated  to  test  according  to  functional  as 
well  as  design  criteria,  and  should  be  directed  at  performance  over 
a  wide  range  of  situations  rather  than  single  point  performance. 

The  measure  of  system  performance  based  on  operational  aspects  is  a 
more  significant  criterion  for  decision  making  than  mere  technical 
performance  specifications. 

Specification  in  functional  terms  is  more  significant  to  a  decision 
maker  than  design  specification.  As  an  example,  although  it  is  indeed 
possible  to  make  a  prediction  of  acoustic  detection  range  based  on  the 
sonar  equation  (Figure  of  Merit),  there  should  be  a  clear  statement  of 
performance  in  a  specified  environment,  against  a  specified  target  class 
operating  in  a  specified  environment. 

13.  RELIABILITY  AND  MAINTAINABILITY 

Reliability  and  maintainability  goals  should  be  specified  early  in 
the  program. 

Numerical  success  criteria  for  primary  aspects  of  reliability  and 
maintenance  should  be  stated.  A  plan  to  demonstrate  these  criteria  should 
be  formulated,  to  include  testing  under  operational  conditions  with  typical 
operating  and  maintenance  personnel  being  employed.  In  specifying  these 
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goals,  operational  failures  should  be  identified  separately  from  technical 
failures.  As  an  example,  if  an  ASW  acoustic  processor  has  16  channels,  and 
the  system  will  function  adequately  with  fewer  than  16  operating,  it  should 
be  clearly  understood  in  advance  whether  failure  of  one  channel  is  counted 
as  a  reliability  failure. 

14.  DATA  HANDLING 

Overall  system  tests  (sensor  weapon  platform)  should  be  conducted  to 
show  achievement  of  specified  "time-late." 

In  the  case  of  some  sonar  systems,  the  data  processing  will  be  done  at 
a  location  remote  from  the  sensor.  A  demonstration  that  the  volume  of  data 
acquired  by  the  sensor  can  be  successfully  transmitted  to  the  remote  site 
with  acceptable  error  rate;  that  it  can  be  made  secure  from  interception; 
and  that  it  can  be  made  jamming  resistent  should  be  planned.  In  order  to 
minimize  "time-late,"  which  is  crucial  to  4SW  effectiveness,  the  system 
should  provide  for  near  real-time  operation  and  automatic  relay.  Demon¬ 
strations  should  also  include  the  ability  of  human  operators  to  manage 
the  data  as  they  are  received. 

15.  AUTOMATED  DETECTION  SCHEMES 

If  system  effectiveness  depends  on  the  successful  development  of 
automated  detection  schemes,  these  schemes  must  be  demonstrated. 

Test  criteria  associated  with  these  demonstrations  should  be  based  on 
detection  thresholds  and  should  include  acceptable  tradeoffs,  as  a  function 
of  threshold  levels,  between  false  target  rates  and  failure  to  detect  true 
targets. 
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II.  VALIDATION  PHASE 


During  the  Validation  Phase,  solutions  to  the  previously  noted  criti¬ 
cal  questions  and  issues  of  risk  are  Identified.  Examination  of  the  T&E 
plan  should  be  made  in  preparation  for  DSARC  milestone  II,  to  see  if  it 
provides  for  clear,  unambiguous,  valid  tests  to  demonstrate  that  the  pro¬ 
posed  solutions  work.  Involvement  of  the  Independent  Test  Agency  should 
be  apparent  by  this  time. 

DT&E  of  subsystems  and  components  will  be  proceeding,  and  technical 
specifications  will  be  the  subject  of  test.  An  examination  should  be  made 
to  determine  that  the  technical  parameters  of  the  proposed  sensor  or  array 
subsystem  are  based  on  an  acceptable  propagation  model  and  are  based  ultim¬ 
ately  on  operational  requirements.  Assessment  of  the  model  can  be  made 
from  evidence  that  model  predictions  conform  with  experience  on  existing 
systems.  T&E  should  be  planned  to  show  that  the  installation  afloat  will 
not  be  unwieldy  or  unsupportable. 

If  the  proposed  data  transmission/handling  system  requires  advances 
in  state-of-the-art  for  components  or  computer  techniques,  integrated 
tests  should  be  scheduled  early  so  that  fail-backs  can  be  adopted  if  the 
advances  are  unsuccessful.  T&E  criteria  should  include  statements  of 
acceptable  tradeoff  between  missed-contact  and  false-alarm  rates,  and 
tests  designed  for  them.  Further  plans  should  be  made  for  full-system 
IOT&E  including  demonstration  of  automation,  operator  alerting  features, 
and  measurement  of  mean  "time-late."  Target  service  requirements  are 
identified  with  respect  to  numbers  of  tests  required  for  statistical  con¬ 
fidence  in  results.  If  the  proposed  system  is  part  of  a  program  to  upgrade 
a  present  system,  there  should  be  a  provision  for  baseline  measurement  of 
the  existing  system  to  evaluate  improvements  achieved. 

Validation  phase  checklist  items  are  as  follows: 

1.  Building-Block  Testing 

2.  Testing  Under  Various  Sea  Conditions 

3.  Authenticating  the  Human  Factors  Concepts 


Preceding  pege  blank 
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4.  Performance  Measures 

5.  User  Needs  and  Participation 

6.  Software  Testing 

7.  Target  Detection  Testing 

8.  Baseline  Measurements 

9.  Uses  of  Submarine  Targets 

10.  Target  Requirements 

11.  Constraints  on  the  Platform  Resulting  from  Array  Configuration 

12.  Operational  Performance 

The  reader  should  also  review  the  checklist  items  in  the  previous 
phase  since  many  of  them  will  be  applicable  during  this  phase. 
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1.  BUILDING-BLOCK  TESTING 


The  design  of  the  set  of  tests  to  demonstrate  feasibility  prior  to 
DSARC  II  should  be  based  on  a  building  block  concept,  with  high 
technical  risk  Items  being  tested  first  and  with  subsequent  tests 
incorporating  more  of  the  hardware  until  the  complete  system  concept 
has  been  demonstrated  feasible. 


For  example,  if,  in  the  ASW  system,  the  high  risk  item  is  the  sensi¬ 
tivity  of  the  sonobuoy  detector  element,  then  the  demonstration  of  detec¬ 
tor  performance  should  be  conducted  prior  to  a  test  of  the  feasibility 
that  the  detector  element  will  meet  the  more  general  performance  specifi¬ 
cations  placed  upon  the  sonobuoy  subsystem. 

2.  TESTING  UNDER  VARIOUS  SEA  CONDITIONS 

Shipboard  ASW  combat  systems  must  be  designed  uo  operate  effectively 
under  sea  conditions  encountered  in  all  seasons  in  all  the  world's 
oceans. 

It  is  unreasonable,  however,  to  conduct  tests  everywhere  to  validate 
this  capability.  Alternatively,  many  studies  which  have  been  conducted 
in  the  past  and  are  still  on-going  have  produced  a  sufficient  body  of  data 
on  many  of  the  ocean  areas  of  the  world  to  allow  accurate  prediction  of 
parameters  important  to  undersea  warfare.  These  can  and  should  be  used 
early  in  the  program  to  develop  models  for  computer  simulations  to  esti¬ 
mate  the  effectiveness  of  ASW  combat  systems  contemplated  for  ships. 

Adequate  at-sea  testing  should  be  conducted  at  such  point  in  the  program 
as  practicable  to  validate  the  simulations.  The  simulation  technique  is  par¬ 
ticularly  useful  in  comparing  alternative  suites  proposed  for  a  new  class  of 
ship,  although  final  evaluation  must  await  the  availability  of  actual  hardware. 

3.  AUTHENTICATING  THE  HUMAN  FACTORS  CONCEPTS 

At  an  appropriate  time  in  concept  definition  or  development  phase, 

T&E  should  authenticate  the  human  factors  concepts  embodied  in  the 
proposed  system  design,  examining  questions  of  safety,  comfort, 
appropriateness  of  man-machine  interfaces,  as  well  as  the  number 
and  skill  levels  of  the  personnel  required. 
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The  numbers  of  personnel  required  should  be  validated  against  both 
operational  and  maintenance  requirements.  Testing  early  versions  in  the 
"human  acceptability  and  compatibility"  environment  is  extremely  important. 
This  will  also  help  to  validate  the  manning  requirements.  As  an  example, 
in  the  development  of  high-power  active  sonars,  the  effects  of  sound 
pressure  levels  on  crew  comfort  should  be  made  the  subject  of  test  and 
evaluation. 

4 .  PERFORMANCE  MEASURES 

Demonstrations  should  be  designed  to  measure  overall  performance, 
with  statistical  weighting  to  compensate  for  reduced  probabilities 
of  success  at  edge  values  of  condition  parameters. 

Many  development  test  and  evaluation  problems  have  been  intensified  by 
the  contract  form  or  factors  closely  tied  to  the  contract.  Considerable 
attention  is  required  to  ensure  that  the  contract  form  and  provisions  min¬ 
imize  such  problems.  Improper  incentives  can  warp  the  proper  conduct  of 
the  test  and  evaluation. 

In  designing  contractually  required  demonstration  tests,  upon  whose 
outcome  may  depend  large  incentive  payments,  or  even  program  continuation, 
it  is  essential  to  specify  broader  success  criteria  than  simply  target 

I 

detection  or  hit  or  miss  in  a  single  given  scenario.  If  this  is  not  done, 
the  entire  program  may  be  skewed  to  meet  the  requirements  of  the  selected 
scenario,  to  the  detriment  of  exploring  the  entire  performance  envelope. 
Such  factors  as  sea  state,  ocean  environment,  variety  of  targets  and  mer¬ 
chant  ship  interference  may  be  entered  into  success  or  failure  of  the  test. 

5.  USER  NEEDS  AND  PARTICIPATION 

It  is  imperative  that  the  user  participate  in  all  of  the  T&E  phases 
to  ensure  that  the  user  needs  are  represented  in  the  development  of 
the  system  concept  and  hardware. 

This  should  facilitate  the  necessary  communication  and  interaction 
between  the  developing  and  user  command  which  is  especially  needed  during 
the  DT&E  and  IOT&E  phases.  User  input  will  be  particularly  valuable  in 
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the  areas  involving  the  man-machine  interface.  For  example,  the  number  and 
skill  level  of  personnel  assigned  to  maintain  an  ASW  helicopter  in  a  new 
mode  of  use  or  different  ship  platform  is  critical  to  the  success  of  the 
system. 


6.  SOFTWARE  TESTING 

Test  and  evaluation  planners  should  ensure  that  software  products 
associated  with  the  ASW  processor  subsystems  are  tested  appropriately 
during  each  phase. 

Software  has  often  been  developed  more  as  an  add-on  rather  than  as  an 
integral  part  of  the  overall  system.  Software  requirements  need  the  same 
consideration  as  hardware  requirements  in  the  Validation  Phases.  Acceptance 
of  the  developmental  system  from  the  contractor  is  in  large  measure  based 
on  compliance  with  hardware  specifications,  while  software  acceptance  may 
be  based  on  limited  and  selected  demonstrations  by  the  contractor.  This 
may  result  in  acceptance  by  the  government  of  a  system  that  has  not  been 
tested  under  full-load,  or  other  operationally  significant  condition.  As 
an  example,  usual  practices  often  do  not  sufficiently  provide  for  testing 
the  final  operational  software  package  in  OT&E.  In  one  sonar  development 
program,  failure  to  test  this  package  in  conjunction  with  hardware  has 
resulted  in  schedule  delays. 


7.  TARGET  DETECTION  TESTING 

Tests  which  are  designed  to  validate  the  feasibility  of’  a  sonobuoy 
system  to  detect  submarine  targets  must  always  be  conducted  using  the 
full  spectrum  of  signals  (generated  by  the  target  and  the  local 
environmental  conditions  such  as  sea  state,,  shipping,  etc.)  expected 
in  an  operational  scenario. 


Tests  should  be  conducted: 

(a)  Without  the  presence  of  the  true  target,  and  data  collected 
which  relates  the  threshold  setting  of  the  detection  system  to 
the  false  alarm  rate. 

(b)  With  the  true  target  present,  to  collect  data  to  relate  threshold 
setting  to  probability  of  detection  of  the  target. 
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Data  from  the  above  two  tests  should  be  used  to  generate  the  detection 
probabilities  /ersus  false  alarm  rates  curve  which  is  the  main  criterion 
for  exaluating  the  feasibility  of  the  system  to  detect  targets. 


8.  BASELINE  MEASUREMENTS 

When  the  proposed  system  is  a  part  of  a  program  to  upgrade  an  exist¬ 
ing  system,  the  proposed  development  program  should  provide  for  a 
"baseline"  measurement  of  the  existing  system  by  which  to  evaluate 
the  improvements  to  be  achieved  by  the  proposed  system. 

As  an  example,  in  the  case  of  a  new  ASW  combat  system,  which  was 
developed  to  replace  an  older  system,  no  provision  was  made  for  obtaining 
test  data  on  the  predecessor  to  determine  its  baseline  effectiveness,  and 
thus  no  comparison  could  be  made  as  to  the  relative  effectiveness  of  the 
successor.  Later  qualitative  judgments  were  that  the  new  system  did  not 
represent  an  improvement  over  the  old  in  many  respects. 

9.  USES  OF  SUBMARINE  TARGETS 

Test  resources  should  be  wisely  used. 

Since  the  services  of  an  actual  submarine  target  are  difficult  to  come 
by,  the  limited  services  that  are  available  should  be  used  carefully.  How 
much  submarine  service  time  is  proposed  in  the  test  plan?  What  level  of 
confidence  in  the  test  results  will  be  obtainable,  i.e.,  how  many  runs  to 
demonstrate  passive  narrow  band,  active,  etc.?  Are  the  types  of  submarines 
used  truly  representative  of  the  threat?  Full  recognition  should  be  given 
the  fact  that  many  ASW  test  objectives  can'  be  satisfied  without  the  presence 
of  actual  submarines.  Others  may  be  accomplished  using  submarine  target 
simulators.  Still  others,  requiring  actual  submarines,  can  be  achieved  by 
careful  planning  involving  use  of  friendly  submarines  engaged  in  routine 
operations,  e.g.,  transit  from  port-to-port. 

10.  TARGET  REQUIREMENTS 

Target  requirements  should  be  stated  in  the  initial  test  concept,  and 
required  target  development  begun  concurrently  with  that  of  the  basic 
system. 
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Targets  for  test  of  anti-submarine  weapons  pose  special  problems. 
Since  homing  weapon  logic  circuitry  will  probably  contain  provisions  for 
discrimination  between  submarine  and  non-submarine,  the  adapatation  of  any 
submarine  simulator  for  target  use  will  probably  be  inadequate,  in  that 
the  weapon  will  recognize  it  as  a  non-submarine  and  look  elsewhere  for  its 
true  target.  It  might  be  desirable  to  build  test  weapons  with  a  pro¬ 
vision  to  temporarily  deactivate  the  more  sophisticated  logic  circuitry 
for  certain  tests. 

11.  CONSTRAINTS  ON  THE  PLATFORM  RESULTING  FROM  ARRAY  CONFIGURATION 

The  proposed  solutions  should  result  in  an  array  configuration  which 
does  not  result  in  burdensome  constraints  on  the  platform. 


One  should  ask  such  questions  as:  Is  the  platform  restricted  in  man¬ 
euvering?  Are  power  requirements  for  the  system  supportable?  Do  the 
proposed  solutions  result  in  unacceptable  environmental  conditions,  e.g., 
excessive  noise  levels  in  operating/living  spaces  aboard  ship?  In  the 
required  location  of  the  towed  array  winch  such  that  it  will  experience 
extremes  of  shock,  vibration,  salt  water  exposure,  etc.? 

12.  OPERATIONAL  PERFORMANCE 

It  is  desirable  that  the  detection  performance  be  ultimately  speci¬ 
fied  in  operational  terms,  viz.,  a  50  percent  probable  detection 
range  against  a  specified  class  of  target  in  a  specified  environment. 

Until  a  full  system  is  available  for  test,  the  performance  is  usually 
specified  in  terms  of  the  technical  parameters  of  the  system,  e.g.,  direc¬ 
tivity  index,  self  noise,  etc.  The  technical  parameters  of  the  proposed 
system  must  be  predicated  on  an  accepted  propagation  model.  There  should 
be  evidence  that  the  model  predications  conform  to  experience  with  exist¬ 
ing  systems,  i.e.,  how  good  is  the  model?  The  degree  of  uncertainty  to  go 
from  the  technical  parameters  to  the  operational  capabilities  should  be 
identified.  This  should  be  minimized  and  that  which  is  necessary  should 
be  recognized. 


23 


III.  FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


Test  and  evaluation  activity  during  this  period  involves  an  examina¬ 
tion  of  test  results  to  see  if  they  demonstrate  at  an  appropriate  confi¬ 
dence  level  that  specifications  have  been  attained,  and  show  a  reasonable 
probability  that  the  system  considered  for  production  will  meet  the  opera¬ 
tional  need. 

Particular  attention  in  these  examinations  should  be  paid  to  sample 
sizes  and  the  conditions  under  which  the  tests  were  conducted.  In  all 
probability,  tests  will  have  been  conducted  using  scarce  submarine  services, 
and  the  quantity  of  testing  and  type  of  target  may  have  been  more  dictated 
by  target  availability  than  the  needs  of  the  test.  In  some  cases,  for 
example,  a  diesel  submarine  would  not  adequately  represent  a  nuclear  sub¬ 
marine. 

Participation  by  the  independent  test  agency,  OPTEVFOR,  is  important 
throughout  this  phase  because  in  many  cases  DT&E  data  can  later  be  added 
to  the  body  of  OT&E  data  for  expansion  of  the  data  base  and  OPTEVFOR  can 
assist  in  arranging  tests  of  operational  validity  during  DT&E.  An  adequate 
IOT&E  by  OPTEVFOR  is  a  requirement  for  a  decision  regarding  quantity  pro¬ 
duction  of  the  new  system. 

Checklist  items  for  Full-Scale  Engineering  Development  are  as  follows: 

1.  Concurrent  Evaluation  Impact  on  Personnel  Skills 

2.  Target  Realism 

3.  Availability  of  Aircraft  Services 

4.  Acoustic  Propagation  Losses 

5.  Testing  of  Sonar  Domes 

6.  Effect  of  Noise  Interference 

7.  Effect  of  Shipping  Interference 

8.  Demonstration  of  Operational  Suitability 

9.  Human  Factors  in  Data  Processing 

10.  Tactical  Employment  Factors 

11.  Bearing  Accuracy 

Preceding  page  blank 


12.  Multi-mode  Operation 

13.  Mutual  Interference 

14.  VDS  Test  Criteria 

15.  Controlled  Environment 

16.  Target  Noise  Level 

17.  Target  Utilization 

18.  Effect  of  Fixes  on  Prior  Tests 

19.  Testing  Devices 

The  reader  should  also  review  the  checklist  items  in  the  previous 
phases  since  many  of  them  will  be  applicable  in  this  phase. 
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1 .  CONCURRENT  EVALUATION  IMPACT  ON  PERSONNEL  SKILLS 

Whenever  possible  the  IOT&E  should  be  conducted  with  operating  and 
maintenance  personnel  whose  skills  are  considered  typical. 

Shipboard  weapon  systems  are  often  put  through  the  process  of  a  con¬ 
current  evaluation  in  the  course  of  their  acquisition,  wherein  sequential 
DT&E  and  IOT&E,  with  some  overlap,  are  conducted.  Often,  the  same  operat¬ 
ing  and  maintenance  personnel  are  present  and  participate  in  equipment 
installation  and  checkout,  DT&E  and  finally  IOT&E.  In  these  cases,  and 
particularly  if  the  test  program  has  been  long  and  difficult,  a  result  is 
that  IOT&E  is  conducted  with  operating  and  maintenance  personnel  whose 

skills  have  been  peaked  through  long  association  with  the  new  equipment. 
Since  they  may  also  have  been  initially  selected  for  their  aptitude,  they 
can  in  no  sense  be  considered  typical.  Evaluation  of  test  results  must 
take  into  account  the  expertise  achieved  by  these  people  through  long 
association  with  the  T&E  program.  This  situation  occurred  in  the  course  of 
IOT&E  of  an  ASW  torpedo  system. 

2.  TARGET  REALISM 

In  the  engineering  feasibility  sea  test  of  sonobuoys,  and  similar 
sensors,  the  targets  used  (signal  generators  and  conventional  subs) 
should  be  representative  in  that  they  present  the  full  acoustic 
spectrum  associated  with  main  targets  for  which  the  system  was 
designed. 

The  use  of  signal  generators  or  diesel  engine  submarines,  where  the 
principal  targets  are  nuclear,  should  be  discouraged  since  noise  inter¬ 
ference  problems  will  not  be  fully  identified  or  investigated. 

3.  AVAILABILITY  OF  AIRCRAFT  SERVICES 

ASW  systems  using  airborne  processing  equipment  and  sonobuoys  require 
that  the  schedule  of  the  testing  program  account  for  the  aircraft 
availability. 
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Since  the  availability  of  the  aircraft  may  be  a  driving  factor  rela¬ 
tive  to  scheduling  of  DT&E  and  IOT&E  activities,  the  scheduling  of  proc¬ 
essors  and  the  sonobuoys  should  be  coordinated  with  that  of  the  aircraft. 
On  one  program  this  aspect  caused  increased  cost  and  delays  in  testing. 

4.  ACOUSTIC  PROPAGATION  LOSSES 

The  total  propagation  losses  of  an  active  sonar  system  should  be 
identified  during  the  early  engineering  development  test  program. 

Losses  occur  because  of  reverberation,  spreading,  absorption,  image 
interference,  surface  and  bottom  interaction,  and  sound  velocity  profile 
which  are  all  related  to  the  depth  of  the  hydrophone.  If  these  losses  are 
unacceptable,  major  changes  could  be  required  in  the  design  of  various 
system  components.  Fixes  late  in  the  program  may  be  very  expensive. 

5.  TESTING  OF  SONAR  DOMES 

In  the  development  testing  of  a  sonar  designed  as  a  subsystem  on  a 
ship,  the  problem  of  self-noise  resulting  from  the  sonar  dome  should 
be  fully  investigated  since  if  this  is  found  to  be  excessive,  large 
costs  and  time  delays  will  result  until  new  dome  designs  are  made 
available. 

Environmental  effects  on  sonar  domes  should  be  tested  and  evaluated 
before  the  domes  are  accepted  as  part  of  the  sonar  system  concept.  Exces¬ 
sive  pitting  and  salt  water  corrosion  can  cause  degradation  in  sound  trans¬ 
mission  and  increases  in  self-noise  which  would  require  frequent  off- 
station  maintenance. 

6.  EFFECT  OF  NOISE  INTERFERENCE 

In  testing  of  sonobuoys,  noise  interference  problems  should  be  identi¬ 
fied  and  investigated  early. 

The  design  must  take  into  account  such  noise  interference.  Generally 
there  are  three  noise  sources  that  are  generated:  by  the  flow  of  water  at 
the  surface  level;  by  strumming  of  the  hydrophone  lines;  and  by  other 
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shipping  in  the  general  area.  Failure  to  investigate  the  effects  of  these 
on  the  system  could  lead  to  serious  shortcomings  requiring  design  changes. 
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7.  EFFECT  OF  SHIPPING  INTERFERENCE 

As  soon  as  possible,  the  effects  of  merchant  shipping  interference  on 
the  detection  and  tracking  capability  of  an  ASW  sensor  system  should 
be  isolated  and  its  impact  on  the  tactical  limitations  of  the  system 
should  be  critically  analyzed. 

Specifically,  testing  should  be  devoted  to  investigation  of  the  mer¬ 
chant  shipping  signals  obscuring  submarine  generated  signals  or  biasing 
the  bearing  information  on  the  submarine  because  of  the  manner  in  which 
received  signals  are  processed  in  the  tracking  circuit.  Before  the  system 
is  operationally  tested,  this  aspect  of  the  system  must  be  fully  under¬ 
stood,  so  that  the  data  collected  during  IOT&E  can  be  properly  understood 
and  evaluated. 


8.  DEMONSTRATION  OF  OPERATIONAL  SUITABILITY 

Before  any  operational  tests  for  demonstration  of  operational  suita¬ 
bility  and  effectiveness  are  conducted,  an  initial  or  pilot  phase 
should  be  conducted  with  the  primary  purpose  of  shaking  down  the  test 
plan,  and  briefing  or  training  participants  as  necessary  regarding 
the  instrumentation  concept,  the  data  analysis  plan,  and  other  test 
features. 


This  phase  should  be  conducted  early  enough  that  sufficient  time  is 
available  to  make  the  necessary  changes  to  the  IOT&E  plan  as  dictated  by 
the  results  of  the  pilot  test.  As  an  example,  coordination  should  be 
established  early  with  activities  that  will  provide  submarine  target  ser¬ 
vices  so  that  exact  requirements  for  achievement  of  test  objectives  can 
be  made  known  to  them  and  rehearsed  if  necessary.  As  a  further  example, 
T&E  of  variable  depth  sonar  may  require  the  towed  body  to  be  towed  at 
depths  that  could  be  regarded  as  unsafe  for  submarines  participating  in 
the  tests.  The  test  plan  should  identify  such  constraints  and  propose 
alternative  solutions  to  the  problem. 
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9. 


HUMAN  FACTORS  IN  DATA  PROCESSING 


The  human  factors  aspects  associated  with  the  operation  of  the  data 
processing  function  is  especially  critical  to  the  performance  of  an 
ASW  system  and  should  be  a  primary  objective  of  early  operational 
testing. 

During  IOT&E,  these  aspects,  which  can  degrade  system  performance, 
must  be  identified  early  in  the  operational  tests.  Emphasis  should  be 
placed  on  identifying  potential  human  factors  problems  associated  with: 

•  Being  on  alert  for  long  periods  of  time. 

•  The  process  required  for  detection  verification  and  location  of 
a  target. 

•  The  process  required  for  tracking  the  target. 

•  The  overall  interactions  required,  man/machine  and  man-to-man. 

In  order  to  identify  these  potential  human  factor  problem  areas,  the  all-up 
weaoon  system,  or  a  realistic  simulator  of  the  platform,  processor,  and 
related  systems  needs  to  be  available. 


10.  TACTICAL  EMPLOYMENT  FACTORS 

The  IOT&E  of  acoustic  sensor  systems  should  be  conducted  to  evaluate 
tactical  plans  for  employment  under  varying  oceanographic  considera¬ 
tions  and  target  source  levels. 

With  each  new  system,  the  existing  plans  should  be  re-evaluated  since 
the  tactical  employment  factors  affect  the  operational  utility  and  effec¬ 
tiveness  of  the  system.  For  example,  optimum  submarine  barrier  spacing 
distances  may  be  estimated  therefrom. 


11.  BEARING  ACCURACY 

In  the  IOT&E,  the  bearing  accuracy  of  the  acoustic  sensor  system 
should  be  determined  as  a  function  of  frequency  and  signal-to-noise 
ratio. 


A  controlled  set  of  tests,  using  realistic  targets  and  target  ranges 
(including  convergence  zone)  and  ocean  environment  should  be  used  to 
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determine  the  "best"  values  that  can  be  expected  relative  to  bearing 
accuracy.  These  tests  should  be  followed  by  uncontrolled  (but  instru¬ 
mented)  testing  (obtain  a  bearing  on  the  target  following  a  detection) 
when  free  play  is  allowed.  During  these  tests,  friendly  shipping  and 
submarines,  etc.,  should  be  in  the  environment.  The  average  length  of 
time  required,  as  well  as  the  mean  of  the  bearing  accuracy  should  be 
determined  in  order  to  assess  the  bearing  accuracy  capability. 


12.  MULTI-MODE  OPERATION 

The  IOT&E  of  ASW  sensors  and  weapons  should  be  designed  and  conducted 
so  as  to  require  the  system  to  operate  in  all  of  its  modes,  not  just 
the  primary  mode. 

An  overall  evaluation  of  the  system  requires  that  the  various  backup 
modes  be  exercised  and  investigated  to  determine  such  factors  as  the  degra¬ 
dation  of  the  system  capability  using  backup  modes,  in  response  to  counter¬ 
measures,  because  of  failures  in  the  primary  mode,  etc.  A  measure  of  the 
operational  utility  of  the  system  should  include  the  above  aspect.  As  an 
example,  the  ability  of  a  wire-guided  ASW  torpedo  to  consummate  an  attack 
in  case  of  a  broken  wire  should  be  assessed.  In  addition,  sufficient 
testing  should  be  conducted  to  determine  probability  of  wire  breakage  and 
conditions  under  which  this  is  most  likely  to  occur. 


13.  MUTUAL  INTERFERENCE 

In  the  conduct  of  the  IOT&E  of  ASW  sensors,  scenarios  should  be  used 
which  will  demonstrate  that  mutual  interference  of  the  sensors  does 
not  degrade  the  detection,  localization  or  tracking  capability  of  the 
system. 

For  example,  the  various  sensor  deployment  concepts  should  be  tested; 
if  more  than  one  ASW  system  is  employed  on  a  platform  simultaneously  the 
interaction  of  one  system  with  the  other  should  be  evaluated. 
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14. 


VDS  TEST  CRITERIA 


The  behavior  of  towed  bodies  of  variable  depth  sonar  systems  should 
be  tested  and  evaluated  under  all  ship  maneuvers  and  speeds  likely  to 
be  encountered  in  combat. 

This  should  Include  tests  of: 

Kiting  effect 
Retrieving  the  towed  body 
Gyro  stability 

Heading  stabilization  times  after  maneuvers 
Streaming  depth 

Bearing  and  distance  accuracy  while  under  maneuvers 
Self  noise 


15.  CONTROLLED  ENVIRONMENT 

It  is  important  that  the  installation  of  the  system  for  IOT&E  be  in  an 
environment  controlled  with  no  greater  stringency  than  the  intended 
shipboard  installation. 

Modern  ASW  combat  systems  usually  require  a  controlled  environment, 
i.e.,  dry  air  and  cooling,  for  proper  and  reliable  operation. 


16.  TARGET  NOISE  LEVEL 

Ensure  that  the  range  of  target  noise  levels  are  used  in  the  testing 
of  the  system. 

Specifications  are  keyed  to  mean  target  noise  level,  usually  speci¬ 
fied  in  a  number  of  decibels  at  a  specified  frequency.  The  true  population 
of  target  noise  levels  vary  about  this  mean.  How  well  will  the  system 
perform  against  the  quietest?  Against  the  noisiest? 


17 .  TARGET  UTILIZATION 

Because  of  limited  target  service  there  may  be  a  need  to  combine  DT&E 
with  IOT&E.  *  . . . . .  . . .  ' 
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Were  the  tests  sufficiently  oriented  toward  operational  scenarios  to 
provide  realistic  data?  Was  the  target  "opening"  from  the  sonar  or  was 
it  "closing"  from  some  unknown  range  and  direction?  Was  the  case  of  cross¬ 
ing  targets  properly  considered?  Have  tests  been  planned  without  the 
operator  being  alerted?  Were  "live"  targets  used?  Was  the  target  sub¬ 
marine  restricted  in  any  way  so  as  to  present  a  less  difficult  challenge; 
for  instance  was  the  target  restricted  to  a  "box"  such  as  that  created  by 
instrumentation  limits  in  a  range,  or  limited  in  its  freedom  by  a  conti¬ 
nental  shelf  or  an  island? 


I 


18.  EFFECT  OF  FIXES  ON  PRIOR  TESTS 

Be  sure  that  the  fix  on  previous  tests  have  not  introduced  new  problems. 

There  are  usually  a  series  of  tests  conducted  prior  to  the  DSARC  III 
meeting  (pre-production,  reliability  demonstrations).  If  problems  were 
found  (and  fixed)  in  later  tests,  did  they  go  back  and  revalidate  prior 
tests?  This  is  necessary  to  insure  that  in  "fixing"  the  last  problem  they 
didn't  do  something  which  would  cause  prior  tests  to  fail.  Be  particularly 
alert  to  this  when  dealing  with  software  and  interfaces. 


19.  TESTING  DEVICES 

The  state-of-the-art  of  testing  devices  and  procedures  should  be  con¬ 
sidered  when  devising  the  test  requirements.  T&E  may  be  required  on 
new  test  facilities. 

On  one  program,  the  vibration  and  shock  tests  of  the  hoist  system 
(which  weighs  over  40,000  lbs.)  were  made  at  a  special  facility  which 
apparently  "pushed  the  state-of-the-art."  Mechanical  problems  with  the 
facility  caused  delays.  In  addition,  there  was  some  doubt  as  to  whether 
the  results  were  accurate  and  conclusive  and  whether  this,  in  turn,  caused 
design  redundancy. 
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IV.  SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 

Follow-on  OT&E  will  be  conducted  on  early  production  equipment. 
Depending  on  the  type  of  system  under  development  FOT&E  may  be  conducted 
in  conjunction  with  OT&E  of  the  vessel  upon  which  it  initially  is  installed. 
Considerations  of  testing  realism  apply  equally  during  this  phase. 

Checklist  items  for  Production/Deployment  are  as  follows: 

1.  Acceptance  Testing 

2 .  ASW  FOT&E 

3.  Minimum  Discernible  Signals 

4.  Helicopter  Support  Testing 

5.  Realism  of  Test 

6.  Installation  and  Testing 

7.  Hoist  Subsystems  Tests 

8.  Towed  Bodies  and  Cable  Behavior  Tests 

9.  Self-Noise  Tests 

The  reader  should  also  review  the  checklist  items  in  the  previous 
phases,  especially  the  last  phase,  since  many  of  these  items  will  be 
applicable  during  this  phase. 
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1. 


ACCEPTANCE  TESTING 


Because  of  the  need  to  equip  ships  and  submarines  on  both  coasts, 
acceptance  testing  of  ASW  weapons  may  be  carried  on  at  several 
locations  remote  from  the  factory. 

In  such  cases  the  standards  and  tolerances  for  production  -acceptance 
test  and  evaluation  (PAT&E)  must  be  uniform  and  in  agreement  with  factory 
acceptance  standards.  Where  a  portion  of  the  PAT&E  uses  automatic  test 
equipment  (ATE) ,  the  same  standards  and  tolerances  must  be  programmed  into 
the  ATE. 

2 .  ASW  FOT&E 

Operational  Tests  and  evaluations  of  the  capability  of  the  system 
to  detect  and  locate  targets  must  be  conducted  in  the  type  of  ocean 
conditions  expected  to  be  encountered  in  wartime  conditions. 

Tests  should  be  planned  and  conducted  in  various  sea  states,  in  areas 
of  the  oceans  for  which  the  system  will  encounter  convergence  zone  condi¬ 
tions  and  with  wartime  type  ocean  traffic  including  merchant  shipping,  U.S. 
warships,  and  U.S.  and  allied  submarines  as  well  as  enemy  submarines.  It  is 
especially  important  to  ensure  that  the  test  crew  performing  the  detection 
and  location  function  be  unaware  of  when  and  where  the  submarine  will  appear, 
except  for  the  normal  type  of  alerting  available  through  the  various  appli¬ 
cable  CC&C  systems.  In  fact,  operational  tests  should  be  conducted  when, 
indeed,  the  submarine  is  not  present,  to  determine  the  false  alarm  rate  of 
the  system. 

3.  MINIMUM  DISCERNIBLE  SIGNALS 

In  testing  the  operational  capability  and  suitability  of  acoustic 
sensor  systems,  the  minimum  discernible  signal  should  be  determined 
by  using  various  ocean  areas  where  submarine  targets  are  expected 
to  be  encountered. 

OT&E  should  be  conducted: 

(a) Using  a  controlled  situation  (signal  generators,  adjusted  signal- 
to-noise  ratio  and  known  locations  of  targets); 
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(b)  Using  an  uncontrolled  situation  to  determine  the  degradation  of 
operational  minimum  discernible  signals  over  the  above  determined 
optimal  value.  This  situation  should  simulate  true  operational 
conditions  (such  as  uncertainty  as  to  the  presence  of  the  target, 
its  location,  or  what  the  signal-to-noise  ratio  is).  Further,  the 
human  impact  on  this  factor  should  be  allowed  to  e~ter  the  test. 


4.  HELICOPTER  SUPPORT  TESTING 

Where  appropriate  tests  should  be  conducted  at  a  realistic  operating 
tempo,  helicopter  maintenance  should  be  performed  by  the  same  number 
and  same  technical  experience  level  personnel  as  planned  for  normal 
fleet  operations. 


Adaptation  of  helicopters  to  operate  as  ASW  vehicles  based  aboard 
small  ships  poses  special  problems  relative  to  supportability.  Demonstra¬ 
tion  of  adequacy  of  manning,  spare  parts  inventory,  documentation,  and 
helicopter  availability  should  be  major  test  criteria. 


5.  REALISM  OF  TEST 

During  the  testing  of  ASW  systems,  care  should  be  taken  to  insure  that 
the  particular  type  of  subs  and  their  exact  signatures  are  not  known 
since  this  fact  will  bias  the  test  results. 


The  crew  performing  the  detection,  identification  and  localization 
should  be  given  no  more  information  than  would  be  available  to  operational 
crew  on  station  at  sea. 


6.  INSTALLATION  AND  TESTING 

A  provision  should  be  made  which  requires  the  suppliers  of  complex 
(e.g.,  sonar)  systems  to  participate  in  the  installation  and  testing 
of  their  systems. 

Improper  installation  and  testing  procedures  can  seriously  affect  the 
performance  of  such  systems.  If  the  supplier  has  no  responsibility  for 
installing  and  testing  the  system,  then  he  can  not  be  held  responsible  for 
failures  that  could  be  attributed  to  improper  installation  and  testing. 
Moreover,  the  supplier  is  probably  best  qualified  to  evaluate  the  effect  of 
any  structural  design  changes  required  by  installation  of  any  new  tests 
required  in  interfacing  his  system  with  other  systems. 
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The  result  of  over  a  decade  of  development  has  been  an  IVDS  electronics 
system  which  is  regarded  as  operationally  satisfactory,  highly  reliable,  and 
easy  to  maintain.  However,  there  are  still  problems  with  the  hoist  subsys¬ 
tem  as  currently  installed  in  the  DE-1052.  The  nature  of  these  problems  is 
discussed  in  the  next  section. 

7 .  HOIST  SUBSYSTEMS 

The  hoist  subsystem  of  a  Variable  Depth  Sonar  (VPS)  should  be  tested 
on  the  class  of  ship  on  which  it  is  to  be  installed  using  the  same 
operational  configuration. 

Major  problems  have  occurred  with  the  hoist  subsystems  due  to  water 
intrusion  in  the  hoist  room  located  below  decks.  Because  of  this  electri¬ 
cal  shorting,  corrosion  damage,  and  general  deterioration  due  to  inability 
to  maintain  the  hoist  properly  has  resulted.  The  problem  probably  could 
have  been  discovered  in  time  to  take  corrective  actions  if  one  of  the 
preproduction  models  had  been  installed  on  an  existing  ship  of  the  same 
class. 

8.  TOWED  BODIES  AND  CABLE  BEHAVIOR 

Towed  body  and  cable  behavior  of  Variable  Depth  Sonars  should  be 
tested  on  the  class  of  ship  on  which  it  is  to  be  installed,  and  at 
speeds  and  maneuvers  likely  to  be  required  by  operations. 

Tests  have  uncovered  a  number  of  problems  in  the  behavior  of  a  towed 
body.  It  has  been  difficult  to  simulate  the  configuration  of  a  given  class 
of  ship  especially  with  respect  to  the  type  of  wake  "rooster  tail"  which 
affects  the  "kite"  angle  and  retrieval  procedures.  Whenever  possible,  the 
procedures  should  be  developed  by  installing  the  VDS  on  one  of  the  earlier 
ships  of  that  class. 

9.  SELF-NOISE  TESTS 

Ea  iy  self-noise  tests  of  a  VDS  should  be  made  on  the  class  of  ship  on 
which  it  is  to  be  installed  using  the  masking  devices  available  on  the 
ship. 
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The  issue  here  is  how  soon  the  self-noise  noise  configuration  of  the 
ship  and  towed  body  can  be  determined  and  measures  taken  to  minimize  noise 
at  various  speeds.  On  one  program,  a  test  ship  was  used  which  had  a  differ¬ 
ent  pump  jet  exhaust  system,  and  a  somewhat  different  Prairie-Masker  system, 
and  as  a  result  the  self-noise  curves  were  different  and  the  experiments 
did  not  reveal  a  subsequent  noise  "hump"  caused  by  pure  radiated  noise 
emitted  by  the  cable  when  used  on  a  ship  for  which  it  was  scheduled.  If 
noise  tests  had  been  conducted  on  early  classes  of  that  ship,  the  remedies 
and  procedures  for  minimizing  noise  would  have  been  developed  sooner. 
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